{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "aaa138f1-ffac-462e-a55d-824489983d6f",
   "metadata": {},
   "source": [
    "# 智能体Agent控制机器人\n",
    "\n",
    "同济子豪兄 2024-9-7"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cbe443ca-348b-46d7-b811-5e8fe9a5a11c",
   "metadata": {},
   "source": [
    "## 导入工具包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "id": "813008d1-f9f4-4e0c-892a-e128d7f7337f",
   "metadata": {},
   "outputs": [],
   "source": [
    "import openvino_genai as ov_genai"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2f4b05d-e198-40d2-8699-3e2fb12c5855",
   "metadata": {},
   "source": [
    "## 载入OpenVINO IR格式的大模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "67db3419-d3e0-4d86-b88e-3634a106ba1f",
   "metadata": {},
   "outputs": [],
   "source": [
    "# device = 'CPU'\n",
    "device = 'GPU'\n",
    "# device = 'NPU'"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "8713cb22-5eb0-4209-a175-0dfdb57fcce2",
   "metadata": {},
   "outputs": [],
   "source": [
    "pipe = ov_genai.LLMPipeline(\"Qwen2-7B-Instruct-int4-ov\", device)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "89ea659b-1b06-470f-902f-38c5899d0d05",
   "metadata": {},
   "source": [
    "## 提示词：智能体Agent编排动作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "id": "1abcb10d-02a3-4c5a-bf41-2842890b6272",
   "metadata": {},
   "outputs": [],
   "source": [
    "robot_order_template = '''\n",
    "你是我的机器人，请你根据我的指令，以json形式输出接下来要运行的对应函数和你给我的回复\n",
    "你只需要回答一个列表即可，不要回答任何中文\n",
    "【以下是所有动作函数】\n",
    "站立：stand()\n",
    "原地踏步：stepping()\n",
    "前进一步：move_forward()\n",
    "后退一步：move_back()\n",
    "向左平移移动一步：move_left()\n",
    "向右平移移动一步：move_right()\n",
    "向左旋转移动：turn_left()\n",
    "向右旋转移动：turn_right()\n",
    "鞠躬：bow()\n",
    "挥手打招呼：wave()\n",
    "扭腰：twist()\n",
    "捶胸庆祝：celebrate()\n",
    "下蹲：squat()\n",
    "踢右脚：right_shot()\n",
    "踢左脚：left_shot()\n",
    "仰卧起坐：sit_ups()\n",
    "佛山叶问的咏春拳：wing_chun()\n",
    "前倾跌倒起立，也就是从趴下到站起来：stand_up_front()\n",
    "后仰跌倒起立，也就是从躺下到站起来：stand_up_back()\n",
    "巡线跨栏模式，顺着黑色线前进并跨越台阶等障碍物：athletics()\n",
    "播放音乐并跳舞（唱跳RAP）：rap()\n",
    "踢不同颜色的足球：kickball('red')\n",
    "搬运不同颜色的海绵方块：transport('red green blue')\n",
    "\n",
    "【输出限制】\n",
    "你直接输出json即可，从{开始，以}结束，不要输出包含```json的开头或结尾\n",
    "在'action'键中，输出函数名列表，列表中每个元素都是字符串，代表要运行的函数名称和参数。每个函数既可以单独运行，也可以和其他函数先后运行。列表元素的先后顺序，表示执行函数的先后顺序\n",
    "在'response'键中，根据我的指令和你编排的动作，以第一人称简短输出你回复我的话，不要超过20个字，要求幽默、善意、玩梗、有趣。\n",
    "kickball和transport函数需要用双引号\n",
    "\n",
    "【以下是一些具体的例子】\n",
    "我的指令：请你先鞠个躬，然后挥挥手。你回复：{'action':['bow()', 'wave()'], 'response':'敬个礼挥挥手，你是我的好朋友'}\n",
    "我的指令：先前进，再后退，向左转一点，再向右平移。你回复：{'action':['move_forward()', 'move_back()', 'turn_left()', 'move_right()'], 'response':'你真是操作大师'}\n",
    "我的指令：先蹲下，再站起来，最后做个庆祝的动作。你回复：{'action':['squat()', 'stand()', 'celebrate()'], 'response':'像奥运举重冠军的动作'}\n",
    "我的指令：向前走两步，向后退三步。你回复：{'action':['move_forward()', 'move_forward()', 'move_back()', 'move_back()', 'move_back()'], 'response':'恰似历史的进程，充满曲折'}\n",
    "我的指令：先挥挥手，然后踢绿色的足球。你回复：{'action':['wave()', \"kickball('green')\"], 'response':'绿色的足球咱可以踢，绿色的帽子咱可不戴'}\n",
    "我的指令：先活动活动筋骨，然后把红色和蓝色的海绵方块搬运到指定位置。你回复：{'action':['twist()', “transport('red blue')”], 'response':'我听说特斯拉的人形机器人兄弟们，每天都在干这种活'}\n",
    "我的指令：先踢右脚，再踢左脚，然后搬运海绵方块。你回复：{'action':['right_shot()', 'left_shot()', “transport('red green blue')”], 'response':'让我先活动活动，然后让海绵宝宝们各回各家'}\n",
    "我的指令：别躺着了，快起来，把红色和蓝色方块搬运到指定位置。你回复：{'action':['stand_up_back()', “transport('red blue')”], 'response':'我也想躺平啊，奈何得干活儿'}\n",
    "\n",
    "【我现在的指令是】\n",
    "'''"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "4abc504a-4e50-46b9-88d4-c7c97d6f2e4d",
   "metadata": {},
   "source": [
    "## 函数：智能体Agent编排动作"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "7b72fb23-2f26-4ca6-91eb-1972dcce4ce0",
   "metadata": {},
   "outputs": [],
   "source": [
    "def agent_plan_qwen_ov(question=\"先鞠个躬，再打个招呼，蹲下，最后站起来\"):\n",
    "    prompt_human = robot_order_template + question\n",
    "    prompt_machine = \"<|im_start|>system\\n<|im_end|>\\n<|im_start|>user\\n{}<|im_end|>\\n<|im_start|>assistant\\n\".format(prompt_human)\n",
    "    result = pipe.generate(prompt_machine)\n",
    "    action_plan_json = eval(result.texts[0])\n",
    "    print('【大模型输出】\\n', action_plan_json)\n",
    "\n",
    "    # 获取动作编排\n",
    "    agent_plan_list = action_plan_json['action']\n",
    "    agent_plan_str = str(agent_plan_list)\n",
    "    # print('【智能体Agent编排动作列表】\\n', agent_plan_list)\n",
    "    \n",
    "    # 获取AI回复\n",
    "    ai_response = action_plan_json['response']\n",
    "    # print('【AI回复】\\n', ai_response)\n",
    "\n",
    "    # 写入txt文件\n",
    "    with open('temp/agent_plan.txt', 'w') as f:\n",
    "        f.write(agent_plan_str)\n",
    "        \n",
    "    return agent_plan_list, ai_response"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "ad207749-f061-439e-a063-ed7fd512705a",
   "metadata": {},
   "source": [
    "## 测试"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 21,
   "id": "18e9a1d9-d368-4f54-bc61-04ca07cfa13e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "【大模型输出】\n",
      " {'action': ['bow()', 'wave()', 'squat()', 'stand()'], 'response': '行个礼打个招呼，蹲下再起来，精神抖擞'}\n"
     ]
    }
   ],
   "source": [
    "agent_plan_list, ai_response = agent_plan_qwen_ov()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 22,
   "id": "67109d19-d77a-477a-b97c-829f10317a10",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "【大模型输出】\n",
      " {'action': ['bow()', 'wave()', 'celebrate()'], 'response': '弯腰挥手再庆祝，友谊之光永不灭'}\n"
     ]
    }
   ],
   "source": [
    "agent_plan_list, ai_response = agent_plan_qwen_ov('先鞠个躬，然后打个招呼，最后摆个庆祝的动作')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 23,
   "id": "ab5fa94e-59c6-4e0a-a690-c8799ae6ad0a",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "【大模型输出】\n",
      " {'action': ['wave()', 'rap()', 'athletics()'], 'response': '嗨嗨嗨，唱跳RAP来啦，跟上我的节奏，一起舞动起来！'}\n"
     ]
    }
   ],
   "source": [
    "agent_plan_list, ai_response = agent_plan_qwen_ov('打个招呼，给我表演唱跳RAP，最后顺着黑线走')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 24,
   "id": "27b27616-59cb-4ac6-8a20-e5bfe154c80e",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "【大模型输出】\n",
      " {'action': ['wave()', \"kickball('blue')\"], 'response': '嗨，蓝色的足球，准备好了！'}\n"
     ]
    }
   ],
   "source": [
    "agent_plan_list, ai_response = agent_plan_qwen_ov('打个招呼，踢蓝色的足球')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "id": "939380c6-acf0-4974-adab-da8ad6d10685",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "【大模型输出】\n",
      " {'action': [\"transport('red green')\"], 'response': '红绿灯，我来帮你过马路'}\n"
     ]
    }
   ],
   "source": [
    "agent_plan_list, ai_response = agent_plan_qwen_ov('帮我把红色和绿色的海绵方块搬到指定位置')"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "id": "53e00d78-8a0d-4943-adc5-2b5121120635",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "【大模型输出】\n",
      " {'action': ['stand_up_back()', 'wave()', \"kickball('green')\", \"transport('red blue')\"], 'response': '起来嗨起来，绿色的球请走开，红蓝海绵到位置'}\n"
     ]
    }
   ],
   "source": [
    "agent_plan_list, ai_response = agent_plan_qwen_ov('别躺着了，起来跟大家打个招呼，把绿色的球踢走，然后把红色和蓝色的海绵方块搬运到指定位置')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "7cb1f332-f49e-49e5-805a-d86a8adb51b6",
   "metadata": {},
   "source": [
    "## 第一个Demo：文字指令控制机器人"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "id": "d661a8a9-bce0-40ae-a05d-be0f945052c3",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "【大模型输出】\n",
      " {'action': ['move_forward()', 'move_forward()', 'move_forward()', 'move_back()', 'move_back()', 'wave()', 'bow()'], 'response': '三步向前，两步后退，挥手致意，再深深一礼'}\n"
     ]
    }
   ],
   "source": [
    "agent_plan_list, ai_response = agent_plan_qwen_ov('向前走三步，再向后退两步，挥挥手，再鞠个躬')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "672abb33-5cb1-4916-947b-2f3e221a5a0c",
   "metadata": {},
   "source": [
    "## 语音合成，播放AI回复的声音"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 40,
   "id": "63751c68-3f75-4b53-b7a6-a0e21754a6d5",
   "metadata": {},
   "outputs": [],
   "source": [
    "def tts(ai_response):\n",
    "    \n",
    "    # 写入文件\n",
    "    with open('temp/ai_response.txt', 'w', encoding='utf-8') as f:\n",
    "        f.write(ai_response)\n",
    "        \n",
    "    # 传到开发板\n",
    "    !scp temp/ai_response.txt pi@192.168.31.146:~/TonyPi/OpenVINO/temp/\n",
    "    !ssh pi@192.168.31.146 \"python ~/TonyPi/OpenVINO/utils_tts.py\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 42,
   "id": "193fa354-5766-4032-951d-ae497c3b13b4",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "瀵煎叆璇�闊冲悎鎴愭ā鍧�\n",
      "TTS Start\n"
     ]
    }
   ],
   "source": [
    "tts(ai_response)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "72a1dc43-f4f7-4e11-8fbc-26c6a6ffabf5",
   "metadata": {},
   "source": [
    "## 把动作编排txt文件传到开发板"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "id": "b5ae204e-7811-4589-9e96-1141938530ba",
   "metadata": {},
   "outputs": [],
   "source": [
    "!scp temp/agent_plan.txt pi@192.168.31.146:~/TonyPi/OpenVINO/temp/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e5bd1c05-aea9-432c-9946-738d7064a63d",
   "metadata": {},
   "source": [
    "## AIPC远程触发机器人运动\n",
    "> 注意：如果涉及踢足球、巡线、搬运海绵方块，需在VNC Viewer桌面端运行\n",
    ">```shell\n",
    "> cd TonyPi/OpenVINO\n",
    "> python utils_robot.py\n",
    "> ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "id": "f0eee814-ec96-4e5e-80ba-6aca5911d8f2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actions Step-by-Step\n",
      "Agent Action List: ['move_forward()', 'move_forward()', 'move_forward()', 'move_back()', 'move_back()', 'wave()', 'bow()']\n",
      "Action: move_forward()\n",
      "Action: move_forward()\n",
      "Action: move_forward()\n",
      "Action: move_back()\n",
      "Action: move_back()\n",
      "Action: wave()\n",
      "Action: bow()\n"
     ]
    }
   ],
   "source": [
    "!ssh pi@192.168.31.146 \"python ~/TonyPi/OpenVINO/utils_robot.py\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "22d3c0ac-135f-47e0-b76a-85340822c033",
   "metadata": {},
   "source": [
    "## 第二个Demo：语音指令控制机器人j"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "id": "0ed7144e-91dd-4ec7-88b5-fb17d49d71e2",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "寮�濮� 6 绉掑綍闊�\n",
      "褰曢煶缁撴潫\n",
      "寮�濮嬭��闊宠瘑鍒�\n",
      "璇�闊宠瘑鍒�缁撴灉锛� 鎸ユ尌鎵嬶紝鎷樹釜鎭�锛屽線鍓嶈蛋涓ゆ�ワ紝鍐嶅仛涓�涓�鍋ヨ韩鐨勫姩浣溿��\n",
      "璇�闊宠瘑鍒�缁撴灉宸插啓鍏�txt鏂囦欢\n",
      "瀹屾垚褰曢煶+璇�闊宠瘑鍒�\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Recording WAVE '/home/pi/TonyPi/OpenVINO/temp/speech_record.wav' : Signed 16 bit Little Endian, Rate 16000 Hz, Mono\n"
     ]
    }
   ],
   "source": [
    "# 录音\n",
    "!ssh pi@192.168.31.146 \"python TonyPi/OpenVINO/utils_asr.py\""
   ]
  },
  {
   "cell_type": "markdown",
   "id": "91ff119c-34d8-458b-baf0-18e18832609b",
   "metadata": {},
   "source": [
    "语音识别结果已写入`~/TonyPi/OpenVINO/temp/speech_recognition.txt`文件"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "813842ea-0d13-47c1-88ae-4987f070aeb2",
   "metadata": {},
   "source": [
    "## 读取语音识别结果"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "id": "112ffb64-5e2e-4951-ab75-7a57f7ae966c",
   "metadata": {},
   "outputs": [],
   "source": [
    "!scp pi@192.168.31.146:~/TonyPi/OpenVINO/temp/speech_recognition.txt temp/speech_recognition.txt "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "id": "04caafcc-6aec-40b7-87ae-ebe0ee398f64",
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('temp/speech_recognition.txt', 'r', encoding='utf-8') as f:\n",
    "    speech_result = f.read()"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 37,
   "id": "d2eb001c-ad13-4083-b34b-d24364526f7c",
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "'挥挥手，拘个恭，往前走两步，再做一个健身的动作。'"
      ]
     },
     "execution_count": 37,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "speech_result"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "fa433033-989e-4fa3-9d70-3ee1cf9f7314",
   "metadata": {},
   "source": [
    "## 传入语音识别结果，编排生成动作列表"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 38,
   "id": "0efde993-c102-4110-abbf-70bb573f24f6",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "【大模型输出】\n",
      " {'action': ['wave()', 'bow()', 'move_forward()', 'move_forward()', 'sit_ups()'], 'response': '手舞足蹈，礼尚往来，前进步伐，健康生活'}\n"
     ]
    }
   ],
   "source": [
    "agent_plan_list, ai_response = agent_plan_qwen_ov(speech_result)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "d2fda51a-6614-4865-a0fd-9bb5b7ff600a",
   "metadata": {},
   "source": [
    "## 把动作编排txt文件传到开发板"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 50,
   "id": "8dff099f-73c2-4849-913d-608ca165c454",
   "metadata": {},
   "outputs": [],
   "source": [
    "!scp temp/agent_plan.txt pi@192.168.31.146:~/TonyPi/OpenVINO/temp/"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "3e39bab5-67b6-4c63-8524-099a9dd62a67",
   "metadata": {},
   "source": [
    "## AIPC远程触发机器人运动\n",
    "> 注意：如果涉及踢足球、巡线、搬运海绵方块，需在VNC Viewer桌面端运行\n",
    ">```shell\n",
    "> cd TonyPi/OpenVINO\n",
    "> python utils_robot.py\n",
    "> ```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 51,
   "id": "d3f48632-3bac-4d9b-bc67-96e7245568a5",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Actions Step-by-Step\n",
      "Agent Action List: ['wave()', 'bow()', 'celebrate()']\n",
      "Action: wave()\n",
      "Action: bow()\n",
      "Action: celebrate()\n"
     ]
    }
   ],
   "source": [
    "!ssh pi@192.168.31.146 \"python ~/TonyPi/OpenVINO/utils_robot.py\""
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "6abd39ab-c92b-40e8-a19b-1b403fe128b7",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3 (ipykernel)",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.10.14"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
